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Abstract: This paper examines the contribution of focus groups in evaluating learner satisfaction with a Virtual 
Learning Environment (VLE). It explores the views of a group of introductory level Post Compulsory Education 
learners that have a history of disaffection, impoverished learning and challenged written and communication 
skills. The outcome of this study will be used to inform future VLE material design for inclusion in a School policy 
document. Additionally, the findings will contribute to the development of both a broader range of discrete ICT 
programs delivered by a VLE and embedded ICT within a range of vocational qualifications across the Post 
Compulsory Education Vocational Curriculum. 
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1. Introduction 

The use of a Virtual Learning Environment (VLE) in Post Compulsory Education in Further Education 
Colleges (FE) has been increasing incrementally over the last five years. Having begun life in 
Universities and the more ‘traditional’ higher education institutions, VLEs are flexible, accessible and 
encourage the development of communities of practice. They encourage group activities, peer 
support and electronic delivery of learning but are not intrinsically designed to aid those learners at 
the lower end of the academic spectrum. This paper presents a case study of learners on an 
introductory (Level 1) FE course in ICT ascertaining their level of user satisfaction with a VLE. The 
outcome of this study will be used to inform future VLE material design for inclusion in a School policy 
document. Additionally, the findings will contribute to the development of both a broader range of 
discrete ICT programs delivered by a VLE and embedded ICT within a range of vocational 
qualifications across the Post Compulsory Education Vocational Curriculum. 

Focus groups were chosen as the method of data collection for this study, based on work by Morgan 
(1988), using a group of learners whose academic backgrounds are similar in qualification and 
educational history and because they ‘generate hypotheses that derive from the insights and data 
from the group’ (Morgan 1988, Krueger 1988). Focus groups have been in evidence since the 1920s. 
At that time, they were in the guise of survey questionnaires related mainly to products and the 
customer requirements of a product. During World War II and up until the 1970s, focus groups were 
used for market research to elicit wants and needs. From the 1980s onwards, the use of focus groups 
has been used mainly in the health arena and in examining social issues. Since then social scientists 
and program evaluators have found focus groups to be useful in understanding how or why people 
hold certain beliefs about a topic or program of interest. Krueger and Casey (2000) identified that 
focus groups can be used for program development and evaluation, planning, and needs 
assessment. Powell and Single define a focus group as ‘a group of individuals selected and 
assembled by researchers to discuss and comment on, from personal experience, the topic that is the 
subject of the research’ (1996) and rely on ‘interaction within the group based on topics that are 
supplied by the researcher’ (Morgan 1997) confirming that focus groups are an ideal opportunity to 
elicit information from learners in a safe, non-threatening environment (Krueger 1988). Morgan (1998, 
pp58) says that ‘the conversations in focus groups can be a gold mine of information about the ways 
that people behave and the motivations that underline these behaviours.’ 

David Morgan’s book The Focus Group Guidebook’ (1998) has been used throughout this study as 
the empirical work containing the methodological approach and the validated data-gathering 
instrument for the Focus Groups. When designing the VLE interface, factors such as navigatability, 
learning activities and resources for use within the computer-mediated environment (as is a VLE) 
were instrumental in the learner satisfaction of the VLE. The work of Robert Gagne (1965) is critical in 
any study using computers for learning and illustrates the importance of his Instructional Design (ID) 
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to an underlying theory of computer-based learning. Gagnes theory stipulates that there are several 
different types or levels of learning. The significance of these classifications is that each different type 
requires different types of instruction. Gagne identifies five major categories of learning: verbal 
information, intellectual skills, cognitive strategies, motor skills and attitudes (web link accessed on 
18/06/06). Gagne’s Nine Events of Instruction (a sequence of learning events borne out of his ID 
theory) addresses learners in terms of the logical steps that are mapped to the way they learn. The 
events are; gain attention, inform the learner of the objective, stimulate their recall of prior learning, 
present the stimulus, provide learner guidance, elicit performance, give feedback, assess 
performance and enhance retention and transfer of learning. Using this premise, designing a VLE with 
learners for whom written communication is difficult and numerical skills underdeveloped, requires 
‘constant revising and adjusting of our uses of technology to better meet the needs of the program 
and our students' (Bucci et al 2003), yet is ideally matched to Gagne’s theory of developing computer- 
based learning using his ID to change the capabilities of the learners. 

By the very nature of the flexibility that can be achieved in designing a VLE, a vast range of stimulus 
can be embedded and these will address the learners preferred learning style and learning 
processes, which are not completely understood, and are different in detail from one person to 
another (Bostock 2005). This is clearly evidenced in this case study and its learners, where a detailed 
analysis of pre-VLE knowledge was essential as well as a breakdown of the learning style of the 
learners so as to maximise their achievement and success against specified learning objectives. The 
materials available on the VLE for this case study only mapped as far as Instructional Event 6 - Elicit 
performance (practice). Future studies are planned whereby the full set of Instructional Events is 
included in all ICT courses delivered via a VLE. An intrinsic part of the study was the collection of data 
from pre- and on-course diagnostic tests of the learners in the study particularly their literacy and 
numeracy level (www.keyskills4u.com), preferred learning style (www.vark-learn.co.uk) and a review 
of their educational history (Individual Learning Plan). Results identified that more than 30% 
presented with challenged written and communication skills that would require specialist intervention 
and support throughout their studies. The VARK (Fleming 2001) online multiple-choice learning style 
assessment profiled the learners learning styles as mainly kinaesthetic and auditory. Typical of 
auditory learners would be a preference to attend lectures, listen to speaking and like to read aloud. 
For kinaesthetic learners, their preference is to be ‘hands-on’ with practical activities and watch (and 
be part of) demonstrations. The learners’ prior educational histories contained instances of exclusion 
and disaffection with their 11-16 education, many learners not completing their year 10 or 11 studies. 
Their home postcodes indicating that they live in some of the lower socio-economic housing, mostly 
council owned, with 3 learners living in ‘poor’ housing in southeast London. All learners in the study 
completed a pre-VLE questionnaire (Table 3) that identified their personal profile, general level of 
computing experience (including using the internet/web), concerns they had about using a VLE in 
their learning and were asked to rank a number of learning activities/resources they had previously 
used in their educational history. This was used to inform the design of the interface of the VLE and 
the resources/activities within it. Care was taken in the design process such that there was ease of 
navigation for the learner, and a simple hierarchical structure for the relationship between the course 
elements to encourage the learners to make appropriate connections within or between each 
resource and/or activity. Usability, flexibility and pedagogy attributes were considered at all stages of 
the VLE design as content management shortcomings militate against making any improvements 
once the VLE is ‘live’ especially if those improvements involve structure (Vogel 2006). 

2. Methodology 

This study used a mixed methods data collection approach where agency was given to both 
quantitative (Likert-scale questionnaire) and qualitative data (Glaser and Strauss 1967) thereby 
triangulating the results. Creswell (2003, p208) makes clear that the perceived legitimacy with which 
the mixed methods approach is being promoted is expanding. This is supported by detailed reference 
to mixed methods studies in areas as diverse as occupational therapy and AIDS prevention. A new 
‘Handbook of Mixed Methods in the Social and Behavioural Sciences’ (Tashakkori and Teddlie, 2003) 
is cited within Creswell (ibid) as the foremost publication in the field of mixed methods research. In 
this case study, 39 students were actively recruited and all completed the pre-VLE questionnaire 
before being exposed to the VLE itself. The issue of ethics is of paramount consideration in a focus 
group study. Informed consent must be gained prior to any focus group activity in either written or oral 
form. The participants must be told of the consequences of the research and care must be taken to 
reduce any harmful effects of the research on the participants. The usefulness of the research cannot 
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be under-estimated and this must be brought to bear on the participants in terms of their benefit and 
that they can get involved in the change that will result from the study findings. Learners in this study 
signed an agreement that gave permission for transcription and hard copy storage of the focus group 
discussions. All participants in the study had sight of the transcripts at the final stage before coding 
and were able to discuss changes they felt needed to made where meaning or incorrect transcription 
had occurred. Whilst the learners knew each other as a group on the same ICT Program, the focus 
group selection process ensured that the ‘friendship groups’ that existed were separated to allow full 
and frank discussions to take place (Krueger and Casey 2000). Focus groups involve not only ‘vertical 
interaction’ or interaction between the moderator and the interviewees, but also ‘horizontal interaction’ 
among the group participants (Denzin and Lincoln 2003). Data emerges from the interaction of the 
learners in the focus groups, in a language that is native to them alone (Fine 1994). This then has to 
be delicately decoded to elicit the themes contained therein. Kitzinger (1994) explains that the group 
situation creates the group’s own hierarchy of importance, their own words and language. 

Focus groups are a collectivistic method as defined by Denzin and Lincoln (2003) based on 
theoretical and methodological considerations. For example; consideration has to be given to how 
many groups will be held? How many people will be involved in each group? The focus group is an 
unstructured interview guide with introductory questions. There is an overt need to state that there are 
no ‘right’ or ‘wrong’ answers. That it is perfectly acceptable (in fact preferred) if there is disagreement 
on topics. There is much controversy about prior analysis by the researcher of the situation in which 
the subjects have been involved. Merton and Kendall say: ‘Foreknowledge of the situation obviously 
reduces the tasks confronting the investigator, since the interview need not be devoted to discovering 
the objective nature of the situation. Equipped in advance with a content analysis, the interviewer can 
readily distinguish the objective facts of the case from the subjective definitions of the situation’ 
(Merton and Kendall, 1946). In a study by Hart (2001), 8 focus groups were used and looked at the 
experiences and impressions and relationship teenagers in public schools in the United States had 
with computers. The learners were in public school in Florida, Maryland and Illinois. One of the most 
surprising findings was that the learners felt that the quality of their education depended on the 
teacher and not the technology as a better way to learn. This is supported in part by Ainley et al 
(2000), whose studies showed that learners, teachers and parents felt that computers have a positive 
effect on learning. US Research shows that the presence of computers and Internet at home has a 
strong positive association with academic outcomes of school children, particularly children from 
disadvantaged backgrounds (Wilhelm et al 2002). Interestingly, a study by researchers at the National 
Centre for Social and Economic Modelling (NATSEM) found that educational attainment of an 
individual was a stronger predictor of having home computers and the Internet than income (Hellwig 
and Lloyd 2000). 

3. Research design 

The group of 39 learners in this case study exhibited the following characteristics: 

Table 1: Learner characteristics 


Learner Data 

Total number of students 

39 

Males 

35 

Females 

4 

Age (years) 

No of Learners 

16 

27 

17 

8 

18 

4 


Table 2: Learner home postcode 


Home Post code 

DAI 

DA11 

DA8 

DAI 2 

DA2 

BR8 

DAI 5 

DAI 6 

DA9 

SE2 

No. of learners 

9 

6 

4 

4 

3 

3 

3 

2 

2 

3 


The home postcode (zip code) for the learners was spread over a 10 mile radius from the College and 
included areas associated with low-status housing, either local council owned or starter homes as well 
as ‘poor’ housing in south east London (SE2). The pre-VLE questionnaire was aimed at establishing 
the learners’ pre-existing level of computing skill and those chosen for inclusion in question 2 and 3 of 
the questionnaire were explicitly required for the learner studying on the introductory ICT program. 
Learners were asked to score their response to each of the statements as 1 (no experience), 2 (some 
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experience) or 3 (extensive experience). Question 4 was aimed at identifying whether the learner had 
found a prescriptive range of learning activities and resources useful in their learning history. Question 
4 required the learner to respond to each learning activity or resource as 1 (not useful), 2 (some use) 
or 3 (very useful). Question 5 required the learner to respond as 1 (no concerns), 2 (some concerns) 
or 3 (extensive concerns) in respect of their anticipated concern with using a VLE in their learning. 
Statistical Package for the Social Sciences (SPSS) was used to calculate Cronbach's alpha. 
Cronbach's a (alpha) is a quantity defined in multivariate statistics. It has an important use as 
measure of the reliability of a psychometric instrument, since it assesses the extent to which a set of 
test items can be treated as measuring a single latent variable http://en.wikipedia.org/wiki/ 
Cronbach's alpha The Cronbach's a value for the items in question 2 and 3 in the Pre-VLE 
questionnaire were calculated as: 

Q2 Cronbach's a value = 0.791 
Q3 Cronbach's a value = 0.820 

For a set of items to be considered as a scale, the Cronbach's a value must be > 0.7. The results 
obtained here, suggest that the items are measuring the same underlying construct and so have 
"high" or "good" reliability. 

Table 3: Pre-VLE questionnaire 


Question No. 

Question(s) asked/data collected 

1 

General learner profiling information 

Student registration number 

Name 

Address 

Postcode (zip code) 

Gender 

Age at 1 st September 2005) 

2 

General level of learner computing experience 

Burn a CD 

Move files 

Insert clipart image 

Wordwrap text 

Insert a table 

Merge two cells 

Mail Merge 

Animation in Powerpoint 

Insert pictures 

Insert hyperlinks 

3 

Experience of using the internet/web in their learning history 
Send and receive emails 

Research information from the web 

Use MSN or equivalent 

Download a podcast 

Use a webcam 

4 

Usefulness of a learning activity or resource used in previous 
learning history 

Essay writing 

Report writing 

Short answer questions 

Reading text books and/or journal articles 
Accessing/downloading reports/articles in word or pdf format. 
Powerpoint presentations 

Listening to lectures 

Online groups activities 

Chat rooms 

Discussion forum/message boards 

Watching short video clips 

Assessment using online multiple-choice questions 

Gapped Handouts 

Analysing numerical data 

Drawing charts and graphs 

Presentation of work to peers and tutors 
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5 


Concerns that learner had about using a VLE in their learning 


The focus group meetings were held during the learner’s weekly group tutorial sessions (one hour 
long) over a period of six weeks. Rooms were allocated that were comfortable and non-threatening 
and informal debriefs were set up so the participants could wind down post-focus group session. Data 
was collected against the questions asked using notes and then (latterly) recordings that were 
transcribed and coded. Full focus group questions reproduced below. 

Table 4: Focus group questions (based on Morgan 1998, and Vogel 2006) 


Question 

no. 

Question asked/data collected 

i 

What were the learner responses when they looked at the VLE for the 
first time? 

2 

How did the learners find the experience of using the VLE for: 

Curriculum activities? 

User Interface (navigatability)? 

3 

What was the learner perspective on the parts of the VLE raised as not 
so good? 

4 

How has the VLE learning fitted in with other types of learning for the 
program? 

5 

What are the most valuable aspects of the VLE for the learner? (data 
collected on an individual basis). 

6 

Taking into consideration all the discussions so far, please rate learner 
satisfaction of the VLE on a scale of 1 - 5 (1 - will not lead to a chance of 
success on program, 5 - will greatly improve chance of success on the 
program). 


Question 6 required the learner to respond on a 5-point Likert scale. The change from a 3-point Likert 
scale used in the pre-VLE questionnaire was intended to broaden the categories of learners response 
such that learners could differentiate between ‘might lead’ and ‘will definitely lead’, thereby identifying 
their confidence in the VLE as a contributory factor to their success. Learners were asked to score 
their response to the statement in question 6 as 1 (will not lead to a chance of success on the 
program), 2 (might lead to a small chance of success on the program), 3 (might lead to good chance 
of success on the program), 4 (will definitely lead to a small chance of success on the program), 5 
(will greatly improve chance of success on the program). Participants in focus groups often say what 
they do but this might not reflect what they actually do. It is therefore essential to triangulate findings 
using an alternative data collection method to ensure that the data collected is accurate. Fern (2001) 
says that focus groups can be used to supplement other methods, hence the choice of a mixed 
methods approach to this study (see Methodology). McNamara (web link accessed on 01/06/06) 
suggests that the order of a focus group should be ‘Develop the questions, record the responses 
using an audio recorder, reflect the response back to the participant to ensure the correct 
understanding of the response’. This methodology was followed, thereby ensuring the analysis and 
results could be verified and validated with the learner at the earliest opportunity. Researchers 
disagree about the correct number of participants for a successful focus group. Many say that 8-12 
(Kitzinger and Barbour 1999), 6 - 12 (Lindlof 1995), 6 - 8 (Krueger 1988), 5 - 8 (Green and Hart 
1999). Focus groups in this case study were 5-8 strong to get enough groups to derive sufficient 
data to analyse learner satisfaction. 


3.1 Findings 


3.1.1 Questionnaire: 

Learners’ response to question 1 and scored response to questions 2 - 5 for the pre-VLE 
questionnaire were entered into a spreadsheet and analysed using numeric, statistical and graphical 
methods. Learners gave themselves an overall low score in the area of ‘general computing 
experience’ with 80% of learners replying that they had either no experience (47%) or some 
experience (33%) in the computing skills required for studying on the introductory ICT program (figure 
1 ). 

Unsurprisingly, all learners had a high pre-existing level of experience of using the Internet/web in 
their learning history with only 14% of learners replying that they had either no experience (2%) or 
some experience (12%) in the use of the internet/web in their learning (figure 2). 
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Figure 1: Pre-existing level of computer skill experience (Q2) 



Figure 2: Pre-existing level of experience of using the Internet/web in learning history (Q3) 

The usefulness of a range of learning activities/resources resulted in mixed responses. Some learners 
felt that simple activities with answers (gapped handouts, online multiple-choice questions) were 
much more useful than overly long essays/reports that were required in some cases. Other learners 
identified group activities using chat rooms/discussion forums as useful as this was something with 
which they felt they had ‘loads of experience’. Learners scored low in their concerns about using a 
VLE in their learning (figure 3) with 54% having ‘No Concerns’ (21 learners), 33% having ‘Some 
Concerns’ (13 learners) and 13% having ‘Extensive Concerns’ (5 learners). This is not unexpected 
and goes someway to validate the learners self-assessed high level of experience in using the 
internet/web (03). It is suggested that this (apparent) lack of concern with using the VLE has been 
translated by the learner as the learners’ expectation that the VLE will be very similar in use to the 
internet/web and therefore have no concerns for them. 
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Concern in using a VLE in learning 
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Figure 3: Level of concern with using a VLE in learning (Q5) 

A Chi-square test was applied to determine if male and female learners responded differently to each 
of the items in Q2 of the pre-VLE questionnaire. Due to the small sample size of female learners (4), 
the Chi square test returned an error in all cases tested, identifying that the low female count would 
affect the validity of the Chi-square value. This was also the case for each of the items in Q3 of the 
pre-VLE questionnaire. When the Chi-square test was repeated using age at 1 st September, this too 
showed an error in the count for the number of learner’s aged 17 and 18 years. Correlation analysis 
(Pearson Correlation co-efficient r) and t-test analysis were inappropriate as there were no numerical 
values collected as part of the pre-VLE questionnaire (the scores of 1, 2, or 3 are categories or ordinal 
numbers and not true ‘numerical’ values). 

3.1.2 Focus groups: 

Results from analysis of Q6 of the focus group questions indicate that 40% of the learners are 
dissatisfied with the VLE content and scored either 1 (will not lead to improved chance of success on 
the program) or 2 (might lead to a small chance of success on the program). The focus group 
sessions were coded immediately following the session where every line, paragraph, or other section 
or text was coded for relevant themes. As these were identified, they were assigned a working code. 
This meant that definitions were constantly challenged and new codes developed (Glaser and 
Strauss, 1967). Saturation was reached after 26 transcripts were decoded. No new codes were 
required, no new categories emerged and any new transcripts only produced a repetition of themes 
already identified. Themes identified were; ease of navigation in the VLE; clear content; able to revisit 
difficult topics but didn’t like all the on-screen reading; the hot-links were all active; the hot-links were 
to good sites; charts/graphs made understanding numbers easier; audio would have helped with all 
the reading; the chat rooms were ‘cool to hang out in’; it was difficult to get in touch with a tutor and 
that scheduled real-time chats (RTC) with a tutor would have been better; email messaging system 
was a good way of keeping in touch if off sick or during holiday periods; the use of the diary and was 
considered to be useful as a reminder for homework/assignment hand in deadlines and could be used 
for recording all kinds of calendar events (not necessarily academically related ones); the use of 
group-based activities were a good way of improving skills and knowledge by sharing ideas with 
peers; the inclusion of a variety of small activities made the VLE much more interesting to use. 

4. Conclusion 

Overall the case study was considered successful in that it identified pre-existing areas of concern for 
the learners that were to use the VLE. These concerns would be included as part of the redesign brief 
for the VLE interface and learning activities. It also identified that those learners whose predominant 
learning style was either kinaesthetic or auditory, would require different ways in interacting with the 
materials in the VLE. It also confirmed that not all learners’ needs were catered for by developing a 
VLE that was ‘fit for the masses’ and that 40% felt that it would not improve their chance of success 
on the programme. It is acknowledged that the study is limited in that it focussed on a narrow range of 
learners and ICT programs. As a pilot study, whose aim was to ascertain learner satisfaction with a 
VLE, using focus groups as the main data collection instrument, it is felt that the results show that that 
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has been achieved. In the next study, a larger female cohort will be studied as the statistical analysis 
was impeded by the small female representation. 


The use of focus groups in this study has been shown to be a sound method of inquiry by using an 
already validated data-collection instrument and triangulating the results with a quantitative 
questionnaire. Focus groups are ideally suited for small groups where a one-to-one setting can be 
threatening and are most effective where the groups are comfortable, there is no peer pressure and 
intimate topics are not being discussed. They are the data gathering method of choice for use in 
‘plural voice’ situations (Fine 1994) where learners can use their own language and words leading to 
the participant’s involvement as key players in the future development of both a broader range of 
discrete ICT programs delivered by a VLE and embedded ICT within a range of vocational 
qualifications across the Post Compulsory Education Vocational Curriculum. 
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